INTRODUCTION


BACKGROUND

“We are more apt to make a grasping gesture when we speak of grasping an idea than when we speak of grasping a doorknob.” Benjamin Lee Whorf 1

My thesis project is organized around a study that explores the way people conceptualize relations. To do this, we observe the spontaneous gestures people produce while solving reasoning problems called linear syllogisms.2 People rountinely gesture, both in conversation with others and when thinking by themselves.3 Gestures can be metaphorical, representing non-spatial attributes with spatial attributes, and some are analogical, representing non-spatial relations with spatial relations.4 Our key questions are whether people gesture when engaging in relational thinking, as required by the linear syllogisms, and if so, whether the kind of relation being reasoned about will influence the production of gestures? For the purposes of this project, we look at one of the important aspects of the data: gesture production across relation type conditions.5 The group means across these conditions were compared with ANOVAs.


VISUALIZATION INQUIRY

As we saw throughout the course, many choices are made in the course of designing data visualizations, including in scientific contexts. One of the most basic questions pertains to the choice of visual form, which is what I will explore here. In class, we mostly discussed relationships among continuous variables when it comes to this. The data I work with here has a discrete response variable (presense/absence of gesture on a given trial) that I put on a continuous scale as a proportion of trials on which gesture is present. The independent variables or predictors are likewise discrete and will be treated as discrete. Here I will focus on two of them: relation type (spatial vs non-spatial) and axis (horizontal vs vertical), with more focus on the former. My main interest for this project is to consider some of the nuances and alternative possibilities for visualizing this information.



DESCRIPTIVE DISTRIBUTION

An important and valuable visualization task involves representing distributions and descriptive statistics of the data. Here I will visually describe gesture production across all participants in this study.

This chart was designed as a ggplot2 object, making use of a wide range of parameters. In line with Tufte’s theory of data-ink ratio minimization, theme_light() was used to strip away the default background fill. Additional chart junk was removed as well, including all x-axis gridlines and the minor y-axis grid lines. The latter was removed in accordance with the findings of Heer and Bostock (2010) about the limits of added accuracy from gridlines. I attempted to enhance the ease of reading bars that are not directly adjacent to the gridlines by slightly reducing their opacity (alpha = .75). Tick marks were maintained along the x-axis to connect the labels with bars in an explicit way, as they do not appear to add to chart junk in a significant way while highlighting the absence of the first three bars. To develop the aesthetics of the chart, I chose a neutral but attactive hex-based color.

Note that this is a bar chart, not a histogram. It was generated with a geom_col layer. Strictly speaking, it is not a distribution, for it shows a value for each participant (proportion of trials during which they produced a gesture), whereas a histogram would instead show the distribution of gesture production as, for example, the number of trials along the x-axis (0 - 40) and the number of participants in each bin. Arguably, this chart is more informative. The order of bars was arranged in ascending order to foreground gesture production across participants, enabling comparisons comparable to those of a histogram while also showing individual level statistics. For example, we can see that all but 3 participants (87.5%) gestured on at least one trial. We also see that half of the participants gestured on more than half of the trials, and nearly half (41.7%) gestured on over two-thirds of the trials.

I make some further refinements to this chart, as follows:

For this chart, I incorporated some further design considerations. To enhance the truthfulness of the chart I relabeled the bars to correspond to participant id numbers. After all, the x-axis is a discrete, nominal variable, not to be confused with a binned continuous variable (as a histogram would show) or an ordinal variable. The ordinality here is artificial—a function of the ascending ordering of the participants by magnitude of the y value. Maintaining participant ids prevents this from being obscured. It can be misunderstood even with the explicit x-axis label by implying a false correspondence between participant number (as counted after being arranged, like in the previous chart) and the proportion of trials on which the participant gestured. There is no such correlation in the data.

I also enhanced the insightfulness of the chart with additional two elements. First, I split the groups by the variable of interest, relation type, using the color channel. This enables one to distinguish approximately how much of a given participant’s gesturing occurs in each condition. It also gives a sense of a broader pattern: that gestures are more likely to be produced on trials with the spatial relation type. The second element added to make the chart more informative is a horizontal line representing the mean proportion of trials with gesture, which allows the reader to immediately see that, on average, gestures were elicited in just over half of all trials.

In addition to the aesthetics applied in the previous chart, here I chose colors that are pleasant while discernable to the colorblind reader (based on Color Brewer pallettes). I also adjusted the font type to a serif family, based on Turfte’s claim that they are more readable (as corroborated by some results we reviewed in class).

Finally, we can add an element of interactivity to this plot, enabling further exploration with precise values for each bars segment:



COMPARING GROUP MEANS

Although descriptive, the pseudo-distributions above do not directly compare means, as is most frequently done with bar charts. *** ### BAR CHART

The typical approach to representing a inferential mean comparison results (such as ANOVA) is with a bar graph that includes standard error bars, like so:

In this chart, similar aesthetic considerations are included as the foregoing charts, but now we abandon individual-level statistics in favor of comparing the means of gesture production across our conditions of interest. This graph does show quite clearly that the means between spatial and non-spatial groups are different even when standard error is taken into account (as indicated by the error bars), although comparing these values across axes is morewhat more difficult. Comparisons are aided by the gridlines as well as inclusion of precise mean values.

However, by representing the means this way we lose information about the underlying distribution. This has been an increasingly common criticism of using bar plots to show mean comparisons—they may hide vastly different distributions. 6

The boxplot is sometimes proferred as an alternative that shows mean differences in a way that preserves information about their underlying distribution. Below is a boxplot of the same means generated with geom_boxplot. I also add a layer of points with geom_point in order to more directly show the data constituting the distribution:

While the advantage of boxplots is clear, it is also relative to the data. If the difference between means does not owe much to the difference in distributions, then bar plots seem to be adequate, although perhaps boxplots should be prefered regardless, since they would represent this fact. When it comes to the experiments with boxplots at hand, I have a couple lines of thought. First, with respect to the datapoints, it is not clear if they contribute enough to be included. Although they do show the data directly, and the clustering at the minimum and maximum values is informative, there is also the problem of overlap that suggests alternative ways to present this information may be preferable. On the ggplot2 version, I increased their size, reduced alpha to 0.75, and used jitter in order to better show the number of points in regions where there is overlap.

The second issue is that it is not clear to me how to properly translate what a traditional bar chart represents into the boxplot format. Boxplots, also known as box-and-whisker plots, typically show the median with a horizontal line, not the mean. The lower and upper edges show the 1st and 3rd quartiles, and the endpoints of the whiskers represent the minimum and maximum values. This five-point summary is useful, but it is different from the information represented by the bar chart, which shows the means and standard errors about the means. The box plot can of course be modified to convey this information: the median can be replaced with the mean, and the quartiles can be replaced with the standard errors. Yet this would also remove the distributional information that gives it comparative advantages over the bar plot. Because these representational features of the box plot are highly conventionalized, however, changing what is represented is likely to have the side effect of requiring further explanation that detracts from less contextually dependent encoding. Perhaps an alternative can be found that has the boxplot’s virtue of representing aspects of the distribution and a less context-sensitive way of representing means.


SPLIT VIOLIN

A less well-worn graphical form that may prove to be a useful alternative is the violin plot. Violin plots can display distributions of data (typically as a density distribution) with plenty of remaining room for elements that show mean comparison. While violin plots usually have reflective symmetry about their vertical axis, it is also possible to split the plot along this axis so two of the groups being compared are on either side of the symmetry. I propose that this is a useful feature because asymmetry can act as a high-level visual feature to facilitate comparison. That is, asymmetry may be a good way to signal differences in distribution, even for groups with the same mean, especially given people’s sensitivity to bilateral symmetry.7 The following split violin plot represents the means with a horizontal line within the (density) distributions of the underlying data from which the means are calculated.

Some features of this plot_ly visualization:

  • Kernel density estimation (KDE) affected this visualization by smoothing the curve beyond the range of the data. This was addressed by setting the range.
  • Means set with meanline parameter.
  • Hover enables exploration of precise values.

Future improvements:

  • Visualize a shaded region under the curve to represent standard error.
  • Add static labels to the mean values.

CONCLUSION

There is still plenty of room for improvement, but the split violin plot seems to be a viable alternative form for representing mean comparisons. One shortcoming may be the comparability both within and across groups in a two-way comparison, which is enabled by bar graphs. I suspect this is more difficult because of the visual juxtoposition across space that is required to compare the distribution. However, it does not seem this detracts from the possibility of comparing the means, which is the only thing a bar graph enables in this context. Therefore, even if true, this is only a fault to the extent that the distributions provide additional details that may distract from such comparisons, thereby making them more difficult.8 This is likely to be a matter that is significantly affected by the audience for which the visualization is intended. In this case, the intended audience is one of scientists. Even so, many of the basic visualization design considerations should apply, for scientists, for all their training, are human, and they are likely to be affected by the same perceptual features that affect the layperson (although their experience no doubt plays a role in the facility with which they can interpret visualizations of data, particularly in their domain of expertise).




## R version 3.5.0 (2018-04-23)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS High Sierra 10.13.5
## 
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] bindrcpp_0.2.2     forcats_0.3.0      stringr_1.3.1     
##  [4] dplyr_0.7.5        purrr_0.2.4        readr_1.1.1       
##  [7] tidyr_0.8.1        tibble_1.4.2       tidyverse_1.2.1   
## [10] data.table_1.11.2  wesanderson_0.3.6  cowplot_0.9.2     
## [13] ggpubr_0.1.6       magrittr_1.5       waffle_0.7.0      
## [16] igraph_1.2.1       ggraph_1.0.1       plotly_4.7.1.9000 
## [19] ggsignif_0.4.0     ggplot2_2.2.1.9000 rmarkdown_1.9     
## 
## loaded via a namespace (and not attached):
##  [1] viridis_0.5.1      httr_1.3.1         jsonlite_1.5      
##  [4] viridisLite_0.3.0  modelr_0.1.2       shiny_1.0.5       
##  [7] assertthat_0.2.0   cellranger_1.1.0   yaml_2.1.19       
## [10] ggrepel_0.8.0      Rttf2pt1_1.3.6     pillar_1.2.2      
## [13] backports_1.1.2    lattice_0.20-35    glue_1.2.0        
## [16] extrafontdb_1.0    digest_0.6.15      promises_1.0.1    
## [19] RColorBrewer_1.1-2 rvest_0.3.2        colorspace_1.3-2  
## [22] httpuv_1.4.3       htmltools_0.3.6    plyr_1.8.4        
## [25] psych_1.8.4        pkgconfig_2.0.1    broom_0.4.4       
## [28] haven_1.1.1        xtable_1.8-2       scales_0.5.0      
## [31] tweenr_0.1.5       later_0.7.2        ggforce_0.1.2     
## [34] withr_2.1.2        lazyeval_0.2.1     cli_1.0.0         
## [37] mnormt_1.5-5       mime_0.5           crayon_1.3.4      
## [40] readxl_1.1.0       evaluate_0.10.1    nlme_3.1-137      
## [43] MASS_7.3-50        xml2_1.2.0         foreign_0.8-70    
## [46] tools_3.5.0        hms_0.4.2          munsell_0.4.3     
## [49] compiler_3.5.0     rlang_0.2.0        units_0.5-1       
## [52] grid_3.5.0         rstudioapi_0.7     htmlwidgets_1.2   
## [55] crosstalk_1.0.0    labeling_0.3       gtable_0.2.0      
## [58] reshape2_1.4.3     R6_2.2.2           gridExtra_2.3     
## [61] lubridate_1.7.4    knitr_1.20         udunits2_0.13     
## [64] extrafont_0.17     bindr_0.1.1        rprojroot_1.3-2   
## [67] stringi_1.2.2      parallel_3.5.0     Rcpp_0.12.17      
## [70] tidyselect_0.2.4


METHODS

  • After the experimenter left the room, participants solved 40 linear syllogisms, blocked by trial type (n = 24).
  • Each syllogism was presented for 10 seconds.
  • Participants solved each problem by saying “yes” if true, “no” if false, and “can’t tell” if invalid.
  • Data of participants was coded for response accuracy, gesture presense, and axis of gesturing.

Waffleplots not rendering (shown as image).

  • Gestures tend to be along the same axis as that of the stimulus.

Typical visualization of mean comparison results.

ORDER EFFECTS (chronological)

  • More gestures were produced when spatial relations are presented before abstract relations.

  • However, more gestures were produced in the spatial condition even when non-spatial relations were presented first.




  1. Whorf, B. L. (1944) The Relation of Habitual Thought and Behavior to Language. Etc: A Review of General Semantics, 1(4), 197–215.

  2. Linear syllogisms are solved via transitive inference. One possible form of a 3-term linear syllogism is aRb, bRc, aRc. In our experiment, these forms were expressed in natural language as follows: Siri is better than Alexa. Cortana is worse than Alexa. Is Cortana better than Siri? Our problem set consisted of 40 3-term linear syllogisms that systematically varied in terms of relation, axis, and difficulty.

  3. e.g., Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245–266.

  4. Cooperrider, K., & Goldin-Meadow, S. (2017). When Gesture Becomes Analogy. Topics in Cognitive Science, 9(3), 719–737.

  5. Relations were either spatial (above/below and left/right) or non-spatial (better/worse and earlier/later), and either along the horizontal axis (left/right and earlier/later) or vertical axis (above/below and better/worse).

  6. See Page Piccinini’s criticism here. Page advocates for the box plot alternative because it more clearly shows the distribution while also showing the mean difference. *** ### BOX PLOT

  7. This would be interesting to resolve empirically, distinguishing both graphical forms (bar plot, box plot, and violin plot) and, in the case of violin plots, accuracy of comparisons between two symmetrical plots and one asymmetrical plot. It may also be possible to tease out the role of convention by measuring years of education and majors.

  8. This is another interesting empirical issue.